Processors having virtually clustered cores and cache slices

ABSTRACT

A processor of an aspect includes a plurality of logical processors each having one or more corresponding lower level caches. A shared higher level cache is shared by the plurality of logical processors. The shared higher level cache includes a distributed cache slice for each of the logical processors. The processor includes logic to direct an access that misses in one or more lower level caches of a corresponding logical processor to a subset of the distributed cache slices in a virtual cluster that corresponds to the logical processor. Other processors, methods, and systems are also disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation U.S. application Ser. No. 13/729,579,filed Dec. 28, 2012, entitled as “PROCESSORS HAVING VIRTUALLY CLUSTEREDCORES AND CACHE SLICES”, which is herein incorporated by reference.

BACKGROUND Field

Embodiments relate to the field of processors. In particular,embodiments relate to the field of cache management in multi-coreprocessors.

Background Information

Advances in semiconductor processing and logic design have permitted anincrease in the amount of logic that may be included in processors andother integrated circuit devices. As a result, many processors now havemultiple to many cores that are monolithically integrated on a singleintegrated circuit or die. The multiple cores generally help to allowmultiple threads or other workloads to be performed concurrently, whichgenerally helps to increase execution throughput.

However, the multiple cores may have a downside in terms of longer hitand/or miss latencies to a shared cache. Often, the processors have ashared cache that is physically distributed across the integratedcircuit or die. For example, cache slices of the distributed cache maybe co-located with corresponding cores. Commonly, all of the cores areable to access all of the cache slices. The time for a core to accessdata from a given cache slice generally depends on the distance betweenthe core and the given cache slice. As a result, as the number of coreson the processor increases, the average distance between cores and cacheslices, as well as the average access hit and/or miss latencies, alsogenerally increases. In addition, the multiple or many cores also tendto increase the memory address entropy at memory controllers, which maytend to result in lower effective memory bandwidth.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The invention may best be understood by referring to the followingdescription and accompanying drawings that are used to illustrateembodiments of the invention. In the drawings:

FIG. 1 is a block diagram of an example computer system including anembodiment of a processor.

FIG. 2 is a block diagram of a first detailed example embodiment of aprocessor having virtual clusters.

FIG. 3 is a block diagram of a second detailed example embodiment of aprocessor having virtual clusters.

FIG. 4 is a block flow diagram of an embodiment of a method of usingvirtual clusters.

FIG. 5 is a block diagram of an embodiment of a processor having virtualclusters having virtual sub-clusters.

FIG. 6 is a block diagram of an embodiment of a processor having a firstvirtual cluster with a higher power consumption state and a secondvirtual cluster with a selectively reduced lower power consumptionstate.

FIG. 7 is a block diagram of an embodiment of computer system includinga processor having virtual clusters and a memory storing virtual clusteraware software.

FIG. 8A is a block diagram illustrating both an exemplary in-orderpipeline and an exemplary register renaming, out-of-orderissue/execution pipeline according to embodiments of the invention.

FIG. 8B is a block diagram illustrating both an exemplary embodiment ofan in-order architecture core and an exemplary register renaming,out-of-order issue/execution architecture core to be included in aprocessor according to embodiments of the invention.

FIG. 9A is a block diagram of a single processor core, along with itsconnection to the on-die interconnect network and with its local subsetof the Level 2 (L2) cache, according to embodiments of the invention.

FIG. 9B is an expanded view of part of the processor core in FIG. 9Aaccording to embodiments of the invention.

FIG. 10 is a block diagram of a processor that may have more than onecore, may have an integrated memory controller, and may have integratedgraphics according to embodiments of the invention.

FIG. 11 shown is a block diagram of a system in accordance with oneembodiment of the present invention.

FIG. 12 shown is a block diagram of a first more specific exemplarysystem in accordance with an embodiment of the present invention.

FIG. 13 shown is a block diagram of a second more specific exemplarysystem in accordance with an embodiment of the present invention.

FIG. 14 shown is a block diagram of a SoC in accordance with anembodiment of the present invention.

FIG. 15 is a block diagram contrasting the use of a software instructionconverter to convert binary instructions in a source instruction set tobinary instructions in a target instruction set according to embodimentsof the invention.

DETAILED DESCRIPTION

Disclosed herein are processors having virtually clustered cores andcache slices, methods performed by and/or within the processors, andsystems incorporating one or more of such processors. In the followingdescription, numerous specific details are set forth (e.g., specificnumbers and arrangements of cores and cache slices, specific ways ofgroupings of cores and cache slices into clusters, specificinterconnects, processor configurations, microarchitectural details,logic partitioning/integration details, sequences of operations, types,interrelationships of system components, and the like). However, it isunderstood that embodiments of the invention may be practiced withoutthese specific details. In other instances, well-known circuits,structures and techniques have not been shown in detail in order not toobscure the understanding of this description.

FIG. 1 is a block diagram of an example computer system 100 including anembodiment of a processor 101. In some embodiments, the processor may bea general-purpose processor (e.g., of the type used in server, desktop,laptop, netbook, tablet, smartphone, cell phone, smart television, videogame console, and like computer systems). Alternatively, the processormay be a special-purpose processor. Examples of suitable special-purposeprocessors include, but are not limited to, network processors,communications processors, cryptographic processors, graphicsprocessors, co-processors, embedded processors, and digital signalprocessors (DSPs), to name just a few examples.

The processor represents a physical processor, integrated circuit, ordie. The physical processor may include any desired number of cores,other processing elements, or other logical processors. The illustratedembodiment of the physical processor is a multi-core processor includinga first core 1 102-1 through an Nth core N 102-N, where N may be anyappropriate number, often ranging from two to several hundred, moreoften on the order of tens (e.g., ten to one hundred). The cores arecoupled together by one or more on-die or on-substrate interconnectstructures (e.g., a ring, torus, mesh, other known interconnectstructure, etc.). The core 1 includes an execution pipeline 103-1.Similarly, the core N includes an execution pipeline 103-N. Eachexecution pipeline may represent any known pipeline, and may include,for example, fetch logic, decode logic, allocation logic, renaminglogic, scheduling logic, register access logic, execution logic, writeback logic, and commit logic, or various combinations thereof. The coresmay be either single threaded or multi-threaded. The cores 102 may besymmetric cores, asymmetric cores (e.g., heterogeneous cores), or somemay be symmetric and others asymmetric.

The term core often refers to logic located on an integrated circuitthat is capable of maintaining an independent architectural state (e.g.,an execution state), in which the independently maintained architecturalstate is associated with dedicated execution resources. In contrast, theterm hardware thread often refers to logic located on an integratedcircuit that is capable of maintaining an independent architecturalstate, in which the independently maintained architectural state sharesaccess to the execution resources it uses. When certain resources areshared by an architectural state, and others are dedicated to thearchitectural state, the line between a core and a hardware thread isless distinct. Nevertheless, the core and the hardware thread are oftenviewed by an operating system as individual processing elements orlogical processors. The operating system is generally able toindividually schedule operations on each of the cores, hardware threads,or other logical processors or processing elements. In other words, aprocessing element or logical processor, in one embodiment, mayrepresent any on-die processor logic capable of being independentlyassociated with code, such as a software thread, operating system,application, or other code. While many of the embodiments herein showcores as examples of the logical processors or processing elements, inother embodiments these cores may be replaced by other logicalprocessors or processing elements. Examples of such other logicalprocessors or processing elements include, but are not limited to,hardware threads, thread units, thread slots, process units, contexts,and/or any other logic that is capable of holding state and beingindependently associated with code.

Referring again to FIG. 1, the core 1 includes one or more lower levelcaches 104-1. Similarly, the core N includes one or more lower levelcaches 104-N. In some embodiments, each core may have a single level oflower level cache. In other embodiments, each core may have two or morelevels of lower level cache (e.g., a lowest level of cache closest tothe core and a mid-level cache farther from the core). Each of the oneor more lower level caches may cache recently obtained data and/orinstructions. The pipelines within each core may attempt to access dataand/or instructions within the one or more lower level cache(s) of thecorresponding core.

The core 1 102-1 through the core N 102-N also share a higher levelcache 105. The one or more lower level caches 104 are referred to aslower level because they are closer physically and/or logically to theircorresponding cores (e.g., the execution logic within the cores) thanthe shared higher level cache. In some embodiments, the cores may checkthe one or more lower level caches for data and/or instructions first.Then, if the sought data is not found in the one or more lower levels ofcache, the cores may proceed to check the shared higher level cache. Insome embodiments, the higher level shared cache may represent a lastlevel data and/or instruction cache (LLC). The LLC may represent thelast cache in the memory hierarchy on the processor (e.g., on die).

The processor also includes a first cache coherency aware memorycontroller 108-1 to couple the processor with a first memory 109-1, anda second cache coherency aware memory controller 108-2 to couple theprocessor with a second memory 109-2. In some embodiments, each cachecoherency aware memory controller may include home agent logic that isoperable to perform cache coherency and second memory controller logicthat is operable to interact with the memory. For simplicity, in thepresent description, such a home agent and memory controllerfunctionalities will be referred to as a cache coherency aware memorycontroller. Other embodiments may include fewer or more cache coherencyaware memory controllers. Moreover, while in the illustrated embodiment,the cache coherency aware memory controllers are on-die or on-processor,in other embodiments they may instead be off-die or off-processor (e.g.,as one or more chipset components).

Conventionally, the cores would generally access the first and secondmemories through the first and second cache coherency aware memorycontrollers, as well as through any intervening cores that are coupledbetween the accessing cores and the cache coherency aware memorycontrollers. Accesses through multiple or potentially many interveningcores generally tends to take more time, resulting in greater memoryaccess latencies, than accesses through less or no intervening cores.This generally translates to reduced performance in a conventionalsystem, as well as possibly extra power consumption and heat generation(e.g., due to extra electrical signals, to wake intervening sleepingcores, etc.).

The processor also includes first optional bus interface logic 110 tocouple the processor with an input/output (I/O) controller hub 111 andone or more I/O devices 112. In other embodiments, the I/O controllerhub may be replaced by another type of chipset component, or the I/Ocontroller hub may be integrated on-die. The processor also includessecond optional bus interface logic 113 to couple the processor with oneor more other processors 114 (e.g., in the case of a server or othermulti-socket system). In the illustration, the first and second memories109, the I/O controller hub 111, the I/O device(s) 112, and the one ormore processors 114, are shown in dashed lines to indicate that, in someembodiments, they are system-level components of a system in which theprocessor is deployed, and are not necessarily components of theprocessor.

The shared higher level cache 105 is commonly implemented as adistributed cache including multiple physically distributed cache slices107. For example, a first cache slice 1 107-1 may correspond to, andoften be closely physically located on die with (e.g., co-located with),the first core 1 102-1. Similarly, an Nth cache slice N 107-N maycorrespond to, and often be closely physically located on die relativeto (e.g., co-located with), the Nth core N 102-N. The cache slices areoften in the uncore portion of the processor.

A first core-cache slice interface logic 1 106-1 may interface the coresto the first cache slice 1 107-1. Similarly, an Nth core-cache sliceinterface logic N 106-N may interface the cores to the Nth cache slice N107-N. In some embodiments, each core-cache slice interface logic mayinclude a first core interface logic (e.g., a corebo) and a second cacheslice interface logic (e.g., a cachebo) that work together to provide acore-cache slice interface, although this is not required. The set ofall cache slice interface logic may also be referred to herein as acaching agent. The collective set of cache slice interface logic (e.g.,the set of all cachebo) may also be referred to as a caching agent ofthe processor or physical socket.

As discussed above in the background section, the distributed sharedcache may have a downside of longer shared cache hit and/or misslatencies especially when there are many cores. Conventionally, all ofthe cores of the processor usually have access to all of the cacheslices on the die. The time for a core to access data from a given cacheslice generally depends on the distance between the core and the givencache slice. For example, it will generally take longer for a core toaccess data from a remote cache slice than from a nearby cache slice. Asa result, as the number of cores on the processor increases, the averagedistance between cores and cache slices, as well as the average accesshit and/or miss latencies, also generally increases. In addition, themultiple or many cores generally tend to increase the address entropy atcache coherency aware memory controllers (e.g., different cores may beprocessing different workloads and accessing different pages and/or datastructures in memory) which may tend to result in lower effective memorybandwidth.

In some embodiments, the cores 102 and distributed portions of theshared higher level cache 105 of the same monolithic integrated circuit,die, or semiconductor substrate may be logically or virtuallypartitioned, clustered, or otherwise grouped into two or morepartitions, clusters, or other groups. For example, as shown in theillustrated embodiment, the first core 1 102-1, the first cache slice 1107-1, the first interface logic 1 106-1, and the first cache coherencyaware memory controller 108-1 may be logically or virtually clustered orgrouped into a first virtual cluster 115-1. Similarly, the Nth core N102-N, the Nth cache slice N 107-1, the Nth interface logic N 106-N, andthe second cache coherency aware memory controller 108-1 may belogically or virtually clustered or grouped into a second virtualcluster 115-2. Other analogous components (e.g., other cores, othercache slices, etc.) may also be included in each virtual cluster. Inother embodiments, instead of the cores, other logical processors and/orprocessing elements and their corresponding shared higher level cacheslices may be similarly logically or virtually clustered. In otherembodiments, instead of just two virtual clusters there may be three ormore virtual clusters each having a set of one or more cores and cacheslices.

In some embodiments, when an access (e.g., a memory request) misses inthe one or more lower level caches 104 of a core 102, the access may bedirected initially to only the cache slices 107 of the shared higherlevel cache 105 that are in the corresponding virtual cluster 115. Insome embodiments, the logical or virtual partitioning, clustering, orgrouping may be done to reduce average access hit and/or miss latencies.For example, in some embodiments, the grouping may be done based onphysical proximity. For example, neighboring or physically proximatecores and/or their corresponding cache slices may be preferentially orpredominantly clustered or grouped together. In some embodiments, theaverage distance between cores and/or cache slices within a virtualcluster (e.g., the intra-virtual cluster distance) may be less than theaverage distance between all cores and/or cache slices between virtualclusters and/or within a socket (e.g., the inter-virtual clusterdistance). As another example, in some embodiments, the grouping may bedone based on the coupling of the cores to interconnect structures. Forexample, the grouping may be done to predominantly virtually group coreson a common ring interconnect or other interconnect structure together.

Advantageously, this may help to improve performance by reducing hitand/or miss latencies for accesses to the shared higher level cache 105.For example, when neighboring or proximate cores and their correspondingcache slices are grouped, the accesses may be directed initially to onlythe neighboring or proximate cache slices of the shared higher levelcache in the corresponding virtual cluster. Instead of having to checkall of the cache slices, only those cache slices in the virtual clustermay need to be checked initially. Moreover, this may tend to increasethe likelihood that the accesses are serviced by nearby/proximate cacheslices instead of the accesses needing to travel all the way to thefarthest or most remote cache slices on the die. This may tend to reducethe average distance the accesses need to travel in order to beserviced, which in turn may tend to reduce access hit and/or misslatencies. Moreover, when cores and/or cache slices on the same ringinterconnect or other interconnect structure are grouped together,shorter ring stop hops, or other shorter interconnect accesses onaverage may be achieved as compared to if they were not on the sameinterconnect structures. This also may tend to reduce access hit and/ormiss latencies.

In some embodiments, if the access also misses in the cache slices 107in the corresponding virtual cluster 115, then the access may initiallybe directed only to a cache coherency aware memory controller 108 in thesame corresponding virtual cluster 115. In some embodiments, the cachecoherency aware memory controller may be the one closest to the coresand/or cache slices of that virtual cluster (e.g., closest on average,closest to the majority, or the like.). The cache coherency aware memorycontroller may be operable to snoop or otherwise communicate with othercache coherency aware memory controllers in order to attempt to servicethe request (e.g., determine where the memory address is homed).

Advantageously, this may also help to improve performance by decreasingthe address entropy experienced by the cache coherency aware memorycontrollers. Rather than receiving addresses from all of the cores(e.g., all the different threads or other processes executing thereon),each cache coherency aware memory controllers may only receive addressesfrom a subset of the cores in that memory controllers virtual cluster(e.g., in some cases about half or less of the cores). As a result, eachcache coherency aware memory controller may tend to see less addressentropy from fewer cores serviced over a given period of time. Often,different threads or software processes tend to use different datastructures in memory so reducing the number of different threads orsoftware processes handled by a memory controller generally tends toreduce the randomness or entropy of the memory addresses it receives.This may help to improve the overall efficiency of the adaptive pageopen policy implemented by the cache coherency aware memory controller(e.g., resulting in improved page hit ratios). This in turn may tend toincrease the effective memory bandwidth and/or memory throughput.

In some embodiments, the cache coherency aware memory controller mayinclude logic that is operable to snoop one or more caching agents ofone or more other virtual clusters of the processor (i.e., on the samedie and/or within the same physical socket). In some embodiments, thecache coherency aware memory controller may include logic that isoperable to indicate a particular virtual cluster and/or a particularvirtual socket (e.g., by targeting a particular QPI port) when snoopinga remote caching agent of a different processor (i.e., on a differentdie and/or in a different physical socket). Advantageously, this mayhelp to utilize bus bandwidth efficiently (e.g., utilize QPI bandwidthin a balanced fashion). In some embodiments, the cache coherency awarememory controller may include logic that is operable to filter snoops toremote and/or local caching agents of virtual clusters/sockets using asnoop filter. In some embodiments, the cache coherency aware memorycontroller may include logic that is operable to reduce bus traffic(e.g., QPI bandwidth) and/or the associated power consumption byutilizing fanout snoops when snooping remote caching agents for virtualclusters of a different processor, as opposed to directed snoops thatmay result in multiple snoop messages to the same socket. In someembodiments, a single snoop fanout may arrive at a physical socket(e.g., at a QPI module) and in response to the single snoop fanoutmultiple directed snoops may be performed within the physical socket tomultiple caching agents and/or virtual clusters within the physicalsocket (e.g., the QPI module may send one directed snoop to each of thecaching agents in the physical socket).

In some embodiments, each virtual cluster of a processor (e.g., aphysical socket or die) may have a copy of a cache line. Some otherapproaches only allow one copy of the cache line for the processor.Allowing each virtual cluster of the processor to have a copy of thecache line may offer an advantage if the operating system allocatesmemory in the adjacent cluster or the operating system migrates aprocess from one virtual cluster to another while the memory pages werenot migrated to the cluster where the process presently resides.

In the illustrated embodiment, for simplicity, only two virtual clustersare shown. Other embodiments may include three, four, or more virtualclusters. In one aspect, there may be as many clusters as cachecoherency aware memory controllers if it is desired to include a cachecoherency aware memory controller within each cluster, which is oftenadvantageous but is not required. In some embodiments, each of thevirtual clusters may have the same number of cores and/or cache slices.This may offer certain advantages (e.g., in implementation efficiency).In other embodiments, each of the virtual clusters may have a differentnumber of cores and/or cache slices.

To avoid obscuring the description, a relatively simple processor 101has been shown and described. In other embodiments, the processor mayoptionally include other well-known components, such as, for example, aninstruction fetch unit, an instruction scheduling unit, a branchprediction unit, instruction and data caches, instruction and datatranslation lookaside buffers, prefetch buffers, microinstructionqueues, microinstruction sequencers, bus interface units, second orhigher level caches, a retirement unit, a register renaming unit, otherconventional components, and various combinations thereof. There areliterally numerous different combinations and configurations ofcomponents in processors, and embodiments are not limited to anyparticular combination or configuration. The processor may represent anintegrated circuit or set of one or more semiconductor dies or chips(e.g., a single die or chip, or a package incorporating two or more dieor chips). In some embodiments, the processor may represent asystem-on-chip (SoC) and/or a chip multi-processor (CMP).

FIG. 2 is a block diagram of a first detailed example embodiment of aprocessor 201 having a first virtual cluster 215-1 and a second virtualcluster 215-2. In some embodiments, the processor of FIG. 2 may beincluded in the system of FIG. 1. Alternatively, the processor of FIG. 2may be included in a similar or different system. Moreover, the systemof FIG. 1 may include either the same, similar, or different processorthan that of FIG. 2.

The processor includes eighteen cores and eighteen corresponding (e.g.,co-located) cache slices. Each of these is shown in the illustration asa core/slice. A leftmost eight of the cores/slices are coupled with afirst ring interconnect 220-1. A rightmost ten of the cores/slices arecoupled with a second ring interconnect 220-2. The first and second ringinterconnects are coupled by a first inter-ring connection logic 222-1and a second inter-ring connection logic 222-2. In some embodiments,each of the first and second inter-ring connection logics may be aswitch box (Sbox), or other mechanism for coupling ring interconnects.

Also coupled with the first ring interconnect is an optional first businterface logic 213 and an optional second bus interface logic 210. Anoptional third bus interface logic 224 is coupled with the second ringinterconnect. In some embodiments, the first bus interface logic may beused to couple the processor with one or more other processors.Alternatively, in a single socket system, the first bus interface logicmay be omitted. In some embodiments, the second bus interface logic maybe used to couple the processor with an I/O hub (not shown). In otherembodiments, an I/O controller may be integrated on processor. Examplesof suitable bus interface logic include, but are not limited to, QuickPath Interconnect (QPI), other point-to-point interconnects, and otherinterconnects known in the arts.

A first cache coherency aware memory controller 208-1 is coupled withthe first ring interconnect. A second cache coherency aware memorycontroller 208-2 is coupled with the second ring interconnect. The firstcache coherency aware memory controller may be used to couple theprocessor with a first memory (not shown). The second cache coherencyaware memory controller may be used to couple the processor with asecond memory (not shown). In some embodiments, each of the cachecoherency aware memory controllers includes a home agent including cachecoherency functionality and a memory controller including memoryinterface functionality.

In the illustrated embodiment, the first virtual cluster includes alleight of the cores/slices (i.e., core/slice 0 through core/slice 7)coupled with the first ring interconnect as well as one core/slice(i.e., core/slice 8) coupled with the second ring interconnect. Thesecond virtual cluster includes all remaining nine cores/slices (i.e.,core/slice 0 through core/slice 8) coupled with the second ringinterconnect. In other embodiments, different ways of dividing thecores/slices between the virtual clusters may be used. Notice that, insome embodiments, physically proximal cores/slices and/or cores/sliceson the same interconnect structure may be preferentially orpredominantly (i.e., at least a majority) included in a virtual cluster.Although this is not required, this may offer advantages of reducedhit/miss latencies, as described elsewhere herein.

Notice also that, in some embodiments, an equal number of cores/slicesmay be included in each of the virtual clusters. This is not required,but often offers an advantage of a simpler implementation. Commonly thememory address space of the memory may be mapped across all of the cacheslices. For example, each memory address may be mapped to a cache slicethrough a hash function. For example, the hash of a given memory addressmay always provide the same value which may always map to the same cacheslices. Often the hash function may attempt to approximately evenlydistribute the memory addresses across the cache slices. For example,memory address A may always hash to the binary value 0111, memoryaddress B may always hash to the binary value 0101, and so on. Sincethere are fewer cores/slices in a virtual cluster (e.g., half in theillustrated embodiment), a smaller hash may be used.

Referring again to FIG. 2, each of the cores/slices has a correspondingidentifier as shown within the core/slice. These identifiers may bereferred to as core/slice identifiers and/or ring stop identifiers. Eachidentifier uniquely identifies a corresponding core/slice (i.e., eachcore/slice has a different identifier). For example, the identifiervalue 00000 identifies core/slice 0 of the first virtual cluster, theidentifier value 10000 identifies core/slice 8 of the first virtualcluster, the identifier value 00111 identifies core/slice 3 of thesecond virtual cluster, and so on. As shown, in one possible embodiment,the IDs for the cores/slices may be assigned starting from the lowerleft corner (as viewed) of the leftmost ring and moving around theleftmost ring counterclockwise, and starting from the lower left cornerof the rightmost ring and moving around the rightmost ringcounterclockwise.

Notice that, in some embodiments, the bit-0 (i.e., the rightmost bit) ofeach of the identifiers is always binary zero (i.e., 0) for the firstvirtual cluster and is always binary one (i.e., 1) for the secondvirtual cluster. In some embodiments, this bit-0 having a value ofbinary zero (i.e., 0) may be used as an identifier of the first virtualcluster, and this bit-0 having a value of binary one (i.e., 1) may beused as an identifier of the second virtual cluster. The remaining moresignificant bits of the ID uniquely identify one of the cores/sliceswithin the virtual cluster identified by bit-0. Looking at this anotherway, the cores/slices of one virtual cluster may be evenly numbered(e.g., 0, 2, 4, 6, etc.) and the cores/slices of the other virtualcluster may be oddly numbered (e.g., 1, 3, 5, etc.).

In some embodiments, this may allow the hash of a given memory addressto always map to the same corresponding core/slice in all of the virtualclusters. For example, the hash of a given memory address with a valueof 0111 may always map to core/slice 7 in the first virtual cluster aswell as core/slice 7 in the second virtual cluster. In some embodiments,this may be used to efficiently implement virtual cluster-to-virtualcluster transfers of cache data. For example, if the cache data comesfrom core/slice 7 of the first virtual cluster and has ring stop ID01110, then only bit-0 of the ring stop ID may be changed/flipped toobtain the ring stop ID for the corresponding core/slice 7 of the secondvirtual cluster (i.e., changing from 01110 to 01111). Accordingly, insome embodiments, having the same number of cores/slices in each virtualcluster may simplify determining to which core/slice to forward data toin the other virtual clusters, although this is not required.

FIG. 3 is a block diagram of a second detailed example embodiment of aprocessor 301 having a first virtual cluster 315-1 and a second virtualcluster 315-2. In some embodiments, the processor of FIG. 3 may beincluded in the system of FIG. 1. Alternatively, the processor of FIG. 3may be included in a similar or different system. Moreover, the systemof FIG. 1 may include either the same, similar, or different processorthan that of FIG. 3.

The second detailed example embodiment is similar in many ways to thefirst detailed example embodiment previously described. Unless otherwisespecified or clearly apparent the correspondingly named components mayoptionally have the same or similar characteristics. To avoid obscuringthe description, primarily different and/or additional structures andcharacteristics of the second detailed example embodiment will bedescribed, without repeating all of the similar or same features.

The processor includes twelve cores/slices. A leftmost eight of thecores/slices are coupled with a first ring interconnect 320-1. Arightmost four of the cores are coupled with a second half-ringinterconnect 320-2. The first and second interconnects are coupled by afirst inter-ring connection logic 322-1 and a second inter-ringconnection logic 322-2.

Also coupled with the first ring interconnect is an optional first businterface logic 313 (e.g., to couple with one or more other processors),an optional second bus interface logic 310 (e.g., to couple with an I/Ohub), and a first cache coherency aware memory controller 308-1 (e.g.,to couple with a memory). Examples of suitable bus interface logicinclude, but are not limited to, QPI. Also coupled with the secondinterconnect is a second cache coherency aware memory controller 308-2(e.g., to couple with a memory). In some embodiments, each of the cachecoherency aware memory controllers includes a home agent including cachecoherency functionality and a memory controller including memoryinterface functionality.

In the illustrated embodiment, the first virtual cluster includes six(i.e., core/slice 0 through core/slice 5) of the eight cores/slicescoupled with the first ring interconnect. The second virtual clusterincludes the remaining two (i.e., core/slice 0 and core/slice 1) of theeight cores/slices coupled with the first ring interconnect and all fourof the cores/slices coupled with the second interconnect (i.e.,core/slice 2 through core/slice 5). In other embodiments, different waysof dividing the cores/slices between the virtual clusters may be used.Notice that, in some embodiments, physically proximal cores/slicesand/or cores/slices on the same interconnect structure may bepreferentially or predominantly included in a virtual cluster. Althoughthis is not required, this may offer advantages of reduced hit/misslatencies, as described elsewhere herein. Notice also that, in someembodiments, an equal number of cores/slices may be included in each ofthe virtual clusters. This is not required, but often offers anadvantage of a simpler implementation, as previously described.

Each of the cores/slices has a corresponding core/slice identifierand/or ring stop identifier as shown within the core/slice. Notice that,in some embodiments, the bit-0 (i.e., the rightmost bit) of each of theidentifiers is always binary zero (i.e., 0) for the first virtualcluster and is always binary one (i.e., 1) for the second virtualcluster. In some embodiments, this bit-0 having a value of binary zero(i.e., 0) may be used as an identifier of the first virtual cluster, andthis bit-0 having a value of binary one (i.e., 1) may be used as anidentifier of the second virtual cluster. In other embodiments, if thereare three or more virtual clusters, then two or more bits may be used touniquely identify one of the virtual clusters. In some embodiments, thismay allow the hash of a given memory address to always map to the samecorresponding core/slice in all of the virtual clusters. In someembodiments, this may be used to efficiently implement virtualcluster-to-virtual cluster transfers of cache data.

FIGS. 2-3 are illustrative examples of virtual clusters, although thescope of the invention is not limited to just these examples. Otherembodiments may include other numbers of cores and/or cache slices,other couplings of cores and/or cache slices, other types ofinterconnects, other numbers and couplings of cache coherency awarememory controllers, may omit some of the illustrated components and/orinclude additional components, may group cores and/or slices and/ormemory controllers differently into virtual clusters, may have more thantwo virtual clusters, etc.

FIG. 4 is a block flow diagram of an embodiment of a method 440 of usingvirtual clusters. In some embodiments, the method may be performed bythe processor of any of FIGS. 1-3. Alternatively, the method may beperformed by similar or different processors. Moreover, the processorsof any of FIGS. 1-3 may perform the same, similar, or different methods.The features and optional details of the processors also may apply tothe method, which may be performed by and/or within the processors.

The method includes virtually clustering logical processors into avirtual cluster, at block 441. In some embodiments, each of the logicalprocessors has one or more corresponding lower level caches. The methodalso includes directing an access that misses in one or more lower levelcaches corresponding to a logical processor to a subset of distributedcache slices, of a shared higher level cache, that are in a virtualcluster that corresponds to the logical processor, at block 442.

FIG. 5 is a block diagram of an embodiment of a processor 501 havingvirtual clusters 515 that have virtual sub-clusters 530. The processorincludes first though fourth sets of cores/slices 528-1 through 528-4.The processor also includes first, second, third, and fourth cachecoherency aware memory controllers 508-1 through 508-4. Each of thecores/slices and the cache coherency aware memory controllers may besubstantially as described elsewhere herein aside from differentfeatures mentioned below.

The first and second sets of cores/slices 528-1, 528-2 and the first andsecond cache coherency aware memory controllers 508-1, 508-2 areincluded in a first virtual cluster 515-1. The third and fourth sets ofcores/slices 528-3, 528-4 and the third and fourth cache coherency awarememory controllers 508-3, 508-4 are included in a second virtual cluster515-2. In some embodiments, the first virtual cluster 515-1 is logicallyor virtually divided, partitioned, or clustered into a first virtualsub-cluster 530-1 including the first set of cores/slices 528-1 and thefirst cache coherency aware memory controller 508-1, and a secondvirtual sub-cluster 530-2 including the second set of cores/slices 528-2and the second cache coherency aware memory controller 508-2. Likewise,the second virtual cluster 515-2 is logically or virtually divided,partitioned, or clustered into a third virtual sub-cluster 530-3including the third set of cores/slices 528-3 and the third cachecoherency aware memory controller 508-3, and a fourth virtualsub-cluster 530-4 including the fourth set of cores/slices 528-4 and thefourth cache coherency aware memory controller 508-4.

In some embodiments, the sub-clusters of a virtual partition arepredominantly physically proximately grouped. In some embodiments, thesub-clusters of a virtual partition are part of a same interconnect ringor other interconnect structure 520-1, 520-2. In some embodiments, thecores/slices of one virtual sub-cluster of a given virtual partition areon average closer and/or on average have shorter hit/miss latencies to amemory controller of another virtual sub-cluster of the given virtualpartition than virtual sub-clusters of other virtual partitions of aprocessor. In some embodiments, misses to a cache coherency aware memorycontroller of one virtual sub-cluster of a given virtual cluster may bepreferentially or initially attempted to be serviced only by one or moreother virtual sub-clusters of the same given virtual cluster. This mayhelp to provide improved performance.

This is just one illustrative example of sub-clustering. Otherembodiments may have other numbers of virtual clusters, other numbers ofvirtual sub-clusters, other numbers of memory controllers, etc.

FIG. 6 is a block diagram of an embodiment of a processor 601 having afirst virtual cluster with a higher power consumption state 615-1 and asecond virtual cluster with a selectively reduced lower powerconsumption state 615-2. Each of the virtual clusters includes one ormore cores 602 and one or more cache slices 607. The terms lower andhigher are relative terms that are relative to one another not absoluteterms. Examples of suitable ways to reduce the power consumption stateinclude, but are not limited to, reducing the frequency of the virtualcluster, reducing the voltage of the virtual cluster, gating power tothe virtual cluster, and other approaches known in the arts (e.g.,selectively gating power to portions of the second virtual cluster).

In some embodiments, the power consumption of the second virtual clustermay be selectively reduced based on workload (e.g., current or estimatedfuture). For example, when the workload is not sufficient to keep bothvirtual clusters busy (e.g., less than a threshold amount for example30%), an operating system, virtual machine monitor (VMM), or othersoftware, which may be aware of the virtual clustering of the processor,may migrate or move active pages from a cache coherency aware memorycontroller for the second virtual cluster to that of the first virtualcluster, migrate all threads, processes, or other workloads from thesecond virtual cluster to the first virtual cluster, and thenselectively reduce the power consumption state of the second virtualcluster. For example, the software may gate power to the one or morecores 602, the one or more cache slices 607, the cache coherency awarememory controller of the second virtual cluster, put the correspondingmemory to sleep, or some combination thereof. When the workloadincreases, the second virtual cluster may be brought back to itsprevious higher power consumption state.

FIG. 7 is a block diagram of an embodiment of computer system 700including a processor 701 having virtual clusters 715 and a memory 709storing virtual cluster aware software 740. The virtual clusters includea first virtual cluster 715-1 through an Nth virtual cluster 715-N,where N may be any appropriate number, often ranging from two to on theorder of around ten, or so. Each virtual cluster includes one or morecores 702 and one or more cache slices 707. The virtual cluster awaresoftware 740 is aware of the virtual clusters 715. Examples of suitablevirtual cluster aware software include, but are not limited to, anoperating system, a virtual machine monitor, other privileged software,user-level applications, etc.).

In some embodiments, the virtual cluster aware software may includevirtual cluster aware and Non-Uniform Memory Access (NUMA) awaresoftware 742. NUMA recognizes that memory accesses in systems may benon-uniform. For example, memory access times may depend on the locationof the memory relative to a core and/or the location of the memorycontroller. The processor is commonly connected with a single physicalsocket. The physical socket may represent the physical connector on aboard (e.g., a motherboard) where the processor is connected whendeployed in a system. In some embodiments, logically or virtuallyclustering or grouping the cores and cache slices into the virtualclusters may effectively provide multiple logical or virtual sockets.

In some embodiments, the NUMA aware software may perform virtual clusterlevel NUMA improvements or optimizations. For example, the NUMA awaresoftware may attempt to minimize or at least reduce cross-virtual-sockettraffic and/or cross-physical socket traffic, which may help to achievehigher throughput. In some embodiments, the virtual clusters may betreated by the software as virtual sockets. In some embodiments, when aprocess of a given virtual cluster (e.g., running on a core of thatgiven virtual cluster) needs more memory allocated, the NUMA awaresoftware may allocate the additional memory from a portion of theoverall memory corresponding to that given virtual cluster instead offrom other memory not corresponding to that given virtual cluster. Inaddition, in the event of there being no available memory for the givenvirtual cluster, in some embodiments, the NUMA aware software mayallocate the additional memory from a portion of the overall memorycorresponding to another virtual cluster within the same physical socketinstead of from memory corresponding to other physical sockets.Cache-to-cache transfers within a physical socket generally tend to befaster than cache-to-cache transfers between physical sockets.

Accordingly, in some embodiments, the NUMA aware software may allocatememory in a way that is based on and aware of the virtual clusters inorder to help improve performance. For example, in some embodiments, theNUMA aware software may preferentially or initially allocate memory fora given virtual cluster (or given virtual socket) intra-virtual cluster(or intra-virtual socket), rather than inter-virtual cluster (orinter-virtual socket). In some embodiments, the NUMA aware software maypreferentially or initially allocate memory for a given virtual cluster(or virtual socket) intra-physical socket (e.g., from virtual clusterswithin that physical socket) rather than inter-physical socket (e.g.,from virtual clusters of different physical sockets).

In some embodiments, the virtual cluster aware software may also includevirtual cluster aware power management software 744. The virtual clusteraware power management software may perform power management of avirtual cluster as previously described in conjunction with FIG. 6.

In some embodiments, the features of the processors of FIGS. 6-7 may beincluded in the processors of any of FIGS. 1-3. Alternatively, thefeatures of the processors of FIGS. 6-7 may be included in similar ordifferent processors. Moreover, the processors of any of FIG. 1-3 mayhave the same, similar, or different features than the processors ofFIGS. 6-7.

While distributed shared caches have been disclosed herein, otherembodiments may use centralized shared caches. These embodiments maystill benefit from reduced address entropy at the cache coherency awarememory controllers. While embodiments have shown ring-typeinterconnects, other embodiments may use other interconnects, such as,for example, torus, mesh, point-to-point, multi-drop, or other knowntypes of interconnects. Exemplary Core Architectures, Processors, andComputer Architectures

Processor cores may be implemented in different ways, for differentpurposes, and in different processors. For instance, implementations ofsuch cores may include: 1) a general purpose in-order core intended forgeneral-purpose computing; 2) a high performance general purposeout-of-order core intended for general-purpose computing; 3) a specialpurpose core intended primarily for graphics and/or scientific(throughput) computing. Implementations of different processors mayinclude: 1) a CPU including one or more general purpose in-order coresintended for general-purpose computing and/or one or more generalpurpose out-of-order cores intended for general-purpose computing; and2) a coprocessor including one or more special purpose cores intendedprimarily for graphics and/or scientific (throughput). Such differentprocessors lead to different computer system architectures, which mayinclude: 1) the coprocessor on a separate chip from the CPU; 2) thecoprocessor on a separate die in the same package as a CPU; 3) thecoprocessor on the same die as a CPU (in which case, such a coprocessoris sometimes referred to as special purpose logic, such as integratedgraphics and/or scientific (throughput) logic, or as special purposecores); and 4) a system on a chip that may include on the same die thedescribed CPU (sometimes referred to as the application core(s) orapplication processor(s)), the above described coprocessor, andadditional functionality. Exemplary core architectures are describednext, followed by descriptions of exemplary processors and computerarchitectures.

Exemplary Core Architectures In-Order and Out-of-Order Core BlockDiagram

FIG. 8A is a block diagram illustrating both an exemplary in-orderpipeline and an exemplary register renaming, out-of-orderissue/execution pipeline according to embodiments of the invention. FIG.8B is a block diagram illustrating both an exemplary embodiment of anin-order architecture core and an exemplary register renaming,out-of-order issue/execution architecture core to be included in aprocessor according to embodiments of the invention. The solid linedboxes in FIGS. 8A-B illustrate the in-order pipeline and in-order core,while the optional addition of the dashed lined boxes illustrates theregister renaming, out-of-order issue/execution pipeline and core. Giventhat the in-order aspect is a subset of the out-of-order aspect, theout-of-order aspect will be described.

In FIG. 8A, a processor pipeline 800 includes a fetch stage 802, alength decode stage 804, a decode stage 806, an allocation stage 808, arenaming stage 810, a scheduling (also known as a dispatch or issue)stage 812, a register read/memory read stage 814, an execute stage 816,a write back/memory write stage 818, an exception handling stage 822,and a commit stage 824.

FIG. 8B shows processor core 890 including a front end unit 830 coupledto an execution engine unit 850, and both are coupled to a memory unit870. The core 890 may be a reduced instruction set computing (RISC)core, a complex instruction set computing (CISC) core, a very longinstruction word (VLIW) core, or a hybrid or alternative core type. Asyet another option, the core 890 may be a special-purpose core, such as,for example, a network or communication core, compression engine,coprocessor core, general purpose computing graphics processing unit(GPGPU) core, graphics core, or the like.

The front end unit 830 includes a branch prediction unit 832 coupled toan instruction cache unit 834, which is coupled to an instructiontranslation lookaside buffer (TLB) 836, which is coupled to aninstruction fetch unit 838, which is coupled to a decode unit 840. Thedecode unit 840 (or decoder) may decode instructions, and generate as anoutput one or more micro-operations, micro-code entry points,microinstructions, other instructions, or other control signals, whichare decoded from, or which otherwise reflect, or are derived from, theoriginal instructions. The decode unit 840 may be implemented usingvarious different mechanisms. Examples of suitable mechanisms include,but are not limited to, look-up tables, hardware implementations,programmable logic arrays (PLAs), microcode read only memories (ROMs),etc. In one embodiment, the core 890 includes a microcode ROM or othermedium that stores microcode for certain macroinstructions (e.g., indecode unit 840 or otherwise within the front end unit 830). The decodeunit 840 is coupled to a rename/allocator unit 852 in the executionengine unit 850.

The execution engine unit 850 includes the rename/allocator unit 852coupled to a retirement unit 854 and a set of one or more schedulerunit(s) 856. The scheduler unit(s) 856 represents any number ofdifferent schedulers, including reservations stations, centralinstruction window, etc. The scheduler unit(s) 856 is coupled to thephysical register file(s) unit(s) 858. Each of the physical registerfile(s) units 858 represents one or more physical register files,different ones of which store one or more different data types, such asscalar integer, scalar floating point, packed integer, packed floatingpoint, vector integer, vector floating point, status (e.g., aninstruction pointer that is the address of the next instruction to beexecuted), etc. In one embodiment, the physical register file(s) unit858 comprises a vector registers unit, a write mask registers unit, anda scalar registers unit. These register units may provide architecturalvector registers, vector mask registers, and general purpose registers.The physical register file(s) unit(s) 858 is overlapped by theretirement unit 854 to illustrate various ways in which registerrenaming and out-of-order execution may be implemented (e.g., using areorder buffer(s) and a retirement register file(s); using a futurefile(s), a history buffer(s), and a retirement register file(s); using aregister maps and a pool of registers; etc.). The retirement unit 854and the physical register file(s) unit(s) 858 are coupled to theexecution cluster(s) 860. The execution cluster(s) 860 includes a set ofone or more execution units 862 and a set of one or more memory accessunits 864. The execution units 862 may perform various operations (e.g.,shifts, addition, subtraction, multiplication) and on various types ofdata (e.g., scalar floating point, packed integer, packed floatingpoint, vector integer, vector floating point). While some embodimentsmay include a number of execution units dedicated to specific functionsor sets of functions, other embodiments may include only one executionunit or multiple execution units that all perform all functions. Thescheduler unit(s) 856, physical register file(s) unit(s) 858, andexecution cluster(s) 860 are shown as being possibly plural becausecertain embodiments create separate pipelines for certain types ofdata/operations (e.g., a scalar integer pipeline, a scalar floatingpoint/packed integer/packed floating point/vector integer/vectorfloating point pipeline, and/or a memory access pipeline that each havetheir own scheduler unit, physical register file(s) unit, and/orexecution cluster—and in the case of a separate memory access pipeline,certain embodiments are implemented in which only the execution clusterof this pipeline has the memory access unit(s) 864). It should also beunderstood that where separate pipelines are used, one or more of thesepipelines may be out-of-order issue/execution and the rest in-order.

The set of memory access units 864 is coupled to the memory unit 870,which includes a data TLB unit 872 coupled to a data cache unit 874coupled to a level 2 (L2) cache unit 876. In one exemplary embodiment,the memory access units 864 may include a load unit, a store addressunit, and a store data unit, each of which is coupled to the data TLBunit 872 in the memory unit 870. The instruction cache unit 834 isfurther coupled to a level 2 (L2) cache unit 876 in the memory unit 870.The L2 cache unit 876 is coupled to one or more other levels of cacheand eventually to a main memory.

By way of example, the exemplary register renaming, out-of-orderissue/execution core architecture may implement the pipeline 800 asfollows: 1) the instruction fetch 838 performs the fetch and lengthdecoding stages 802 and 804; 2) the decode unit 840 performs the decodestage 806; 3) the rename/allocator unit 852 performs the allocationstage 808 and renaming stage 810; 4) the scheduler unit(s) 856 performsthe schedule stage 812; 5) the physical register file(s) unit(s) 858 andthe memory unit 870 perform the register read/memory read stage 814; theexecution cluster 860 perform the execute stage 816; 6) the memory unit870 and the physical register file(s) unit(s) 858 perform the writeback/memory write stage 818; 7) various units may be involved in theexception handling stage 822; and 8) the retirement unit 854 and thephysical register file(s) unit(s) 858 perform the commit stage 824.

The core 890 may support one or more instructions sets (e.g., the x86instruction set (with some extensions that have been added with newerversions); the MIPS instruction set of MIPS Technologies of Sunnyvale,Calif.; the ARM instruction set (with optional additional extensionssuch as NEON) of ARM Holdings of Sunnyvale, Calif.), including theinstruction(s) described herein. In one embodiment, the core 890includes logic to support a packed data instruction set extension (e.g.,AVX1, AVX2), thereby allowing the operations used by many multimediaapplications to be performed using packed data.

It should be understood that the core may support multithreading(executing two or more parallel sets of operations or threads), and maydo so in a variety of ways including time sliced multithreading,simultaneous multithreading (where a single physical core provides alogical core for each of the threads that physical core issimultaneously multithreading), or a combination thereof (e.g., timesliced fetching and decoding and simultaneous multithreading thereaftersuch as in the Intel® Hyperthreading technology).

While register renaming is described in the context of out-of-orderexecution, it should be understood that register renaming may be used inan in-order architecture. While the illustrated embodiment of theprocessor also includes separate instruction and data cache units834/874 and a shared L2 cache unit 876, alternative embodiments may havea single internal cache for both instructions and data, such as, forexample, a Level 1 (L1) internal cache, or multiple levels of internalcache. In some embodiments, the system may include a combination of aninternal cache and an external cache that is external to the core and/orthe processor. Alternatively, all of the cache may be external to thecore and/or the processor.

Specific Exemplary In-Order Core Architecture

FIGS. 9A-B illustrate a block diagram of a more specific exemplaryin-order core architecture, which core would be one of several logicblocks (including other cores of the same type and/or different types)in a chip. The logic blocks communicate through a high-bandwidthinterconnect network (e.g., a ring network) with some fixed functionlogic, memory I/O interfaces, and other necessary I/O logic, dependingon the application.

FIG. 9A is a block diagram of a single processor core, along with itsconnection to the on-die interconnect network 902 and with its localsubset of the Level 2 (L2) cache 904, according to embodiments of theinvention. In one embodiment, an instruction decoder 900 supports thex86 instruction set with a packed data instruction set extension. An L1cache 906 allows low-latency accesses to cache memory into the scalarand vector units. While in one embodiment (to simplify the design), ascalar unit 908 and a vector unit 910 use separate register sets(respectively, scalar registers 912 and vector registers 914) and datatransferred between them is written to memory and then read back in froma level 1 (L1) cache 906, alternative embodiments of the invention mayuse a different approach (e.g., use a single register set or include acommunication path that allow data to be transferred between the tworegister files without being written and read back).

The local subset of the L2 cache 904 is part of a global L2 cache thatis divided into separate local subsets, one per processor core. Eachprocessor core has a direct access path to its own local subset of theL2 cache 904. Data read by a processor core is stored in its L2 cachesubset 904 and can be accessed quickly, in parallel with other processorcores accessing their own local L2 cache subsets. Data written by aprocessor core is stored in its own L2 cache subset 904 and is flushedfrom other subsets, if necessary. The ring network ensures coherency forshared data. The ring network is bi-directional to allow agents such asprocessor cores, L2 caches and other logic blocks to communicate witheach other within the chip. Each ring data-path is 1012-bits wide perdirection.

FIG. 9B is an expanded view of part of the processor core in FIG. 9Aaccording to embodiments of the invention. FIG. 9B includes an L1 datacache 906A part of the L1 cache 904, as well as more detail regardingthe vector unit 910 and the vector registers 914. Specifically, thevector unit 910 is a 16-wide vector processing unit (VPU) (see the16-wide ALU 928), which executes one or more of integer,single-precision float, and double-precision float instructions. The VPUsupports swizzling the register inputs with swizzle unit 920, numericconversion with numeric convert units 922A-B, and replication withreplication unit 924 on the memory input. Write mask registers 926 allowpredicating resulting vector writes.

Processor with Integrated Memory Controller and Graphics

FIG. 10 is a block diagram of a processor 1000 that may have more thanone core, may have an integrated memory controller, and may haveintegrated graphics according to embodiments of the invention. The solidlined boxes in FIG. 10 illustrate a processor 1000 with a single core1002A, a system agent 1010, a set of one or more bus controller units1016, while the optional addition of the dashed lined boxes illustratesan alternative processor 1000 with multiple cores 1002A-N, a set of oneor more integrated memory controller unit(s) 1014 in the system agentunit 1010, and special purpose logic 1008.

Thus, different implementations of the processor 1000 may include: 1) aCPU with the special purpose logic 1008 being integrated graphics and/orscientific (throughput) logic (which may include one or more cores), andthe cores 1002A-N being one or more general purpose cores (e.g., generalpurpose in-order cores, general purpose out-of-order cores, acombination of the two); 2) a coprocessor with the cores 1002A-N being alarge number of special purpose cores intended primarily for graphicsand/or scientific (throughput); and 3) a coprocessor with the cores1002A-N being a large number of general purpose in-order cores. Thus,the processor 1000 may be a general-purpose processor, coprocessor orspecial-purpose processor, such as, for example, a network orcommunication processor, compression engine, graphics processor, GPGPU(general purpose graphics processing unit), a high-throughput manyintegrated core (MIC) coprocessor (including 30 or more cores), embeddedprocessor, or the like. The processor may be implemented on one or morechips. The processor 1000 may be a part of and/or may be implemented onone or more substrates using any of a number of process technologies,such as, for example, BiCMOS, CMOS, or NMOS.

The memory hierarchy includes one or more levels of cache within thecores, a set or one or more shared cache units 1006, and external memory(not shown) coupled to the set of integrated memory controller units1014. The set of shared cache units 1006 may include one or moremid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), orother levels of cache, a last level cache (LLC), and/or combinationsthereof. While in one embodiment a ring based interconnect unit 1012interconnects the integrated graphics logic 1008, the set of sharedcache units 1006, and the system agent unit 1010/integrated memorycontroller unit(s) 1014, alternative embodiments may use any number ofwell-known techniques for interconnecting such units. In one embodiment,coherency is maintained between one or more cache units 1006 and cores1002-A-N.

In some embodiments, one or more of the cores 1002A-N are capable ofmulti-threading. The system agent 1010 includes those componentscoordinating and operating cores 1002A-N. The system agent unit 1010 mayinclude for example a power control unit (PCU) and a display unit. ThePCU may be or include logic and components needed for regulating thepower state of the cores 1002A-N and the integrated graphics logic 1008.The display unit is for driving one or more externally connecteddisplays.

The cores 1002A-N may be homogenous or heterogeneous in terms ofarchitecture instruction set; that is, two or more of the cores 1002A-Nmay be capable of execution the same instruction set, while others maybe capable of executing only a subset of that instruction set or adifferent instruction set.

Exemplary Computer Architectures

FIGS. 11-14 are block diagrams of exemplary computer architectures.Other system designs and configurations known in the arts for laptops,desktops, handheld PCs, personal digital assistants, engineeringworkstations, servers, network devices, network hubs, switches, embeddedprocessors, digital signal processors (DSPs), graphics devices, videogame devices, set-top boxes, micro controllers, cell phones, portablemedia players, hand held devices, and various other electronic devices,are also suitable. In general, a huge variety of systems or electronicdevices capable of incorporating a processor and/or other executionlogic as disclosed herein are generally suitable.

Referring now to FIG. 11, shown is a block diagram of a system 1100 inaccordance with one embodiment of the present invention. The system 1100may include one or more processors 1110, 1115, which are coupled to acontroller hub 1120. In one embodiment the controller hub 1120 includesa graphics memory controller hub (GMCH) 1190 and an Input/Output Hub(IOH) 1150 (which may be on separate chips); the GMCH 1190 includesmemory and graphics controllers to which are coupled memory 1140 and acoprocessor 1145; the IOH 1150 is couples input/output (I/O) devices1160 to the GMCH 1190. Alternatively, one or both of the memory andgraphics controllers are integrated within the processor (as describedherein), the memory 1140 and the coprocessor 1145 are coupled directlyto the processor 1110, and the controller hub 1120 in a single chip withthe IOH 1150.

The optional nature of additional processors 1115 is denoted in FIG. 11with broken lines. Each processor 1110, 1115 may include one or more ofthe processing cores described herein and may be some version of theprocessor 1000.

The memory 1140 may be, for example, dynamic random access memory(DRAM), phase change memory (PCM), or a combination of the two. For atleast one embodiment, the controller hub 1120 communicates with theprocessor(s) 1110, 1115 via a multi-drop bus, such as a frontside bus(FSB), point-to-point interface such as QuickPath Interconnect (QPI), orsimilar connection 1195.

In one embodiment, the coprocessor 1145 is a special-purpose processor,such as, for example, a high-throughput MIC processor, a network orcommunication processor, compression engine, graphics processor, GPGPU,embedded processor, or the like. In one embodiment, controller hub 1120may include an integrated graphics accelerator.

There can be a variety of differences between the physical resources1110, 1115 in terms of a spectrum of metrics of merit includingarchitectural, microarchitectural, thermal, power consumptioncharacteristics, and the like.

In one embodiment, the processor 1110 executes instructions that controldata processing operations of a general type. Embedded within theinstructions may be coprocessor instructions. The processor 1110recognizes these coprocessor instructions as being of a type that shouldbe executed by the attached coprocessor 1145. Accordingly, the processor1110 issues these coprocessor instructions (or control signalsrepresenting coprocessor instructions) on a coprocessor bus or otherinterconnect, to coprocessor 1145. Coprocessor(s) 1145 accept andexecute the received coprocessor instructions.

Referring now to FIG. 12, shown is a block diagram of a first morespecific exemplary system 1200 in accordance with an embodiment of thepresent invention. As shown in FIG. 12, multiprocessor system 1200 is apoint-to-point interconnect system, and includes a first processor 1270and a second processor 1280 coupled via a point-to-point interconnect1250. Each of processors 1270 and 1280 may be some version of theprocessor 1000. In one embodiment of the invention, processors 1270 and1280 are respectively processors 1110 and 1115, while coprocessor 1238is coprocessor 1145. In another embodiment, processors 1270 and 1280 arerespectively processor 1110 coprocessor 1145.

Processors 1270 and 1280 are shown including integrated memorycontroller (IMC) units 1272 and 1282, respectively. Processor 1270 alsoincludes as part of its bus controller units point-to-point (P-P)interfaces 1276 and 1278; similarly, second processor 1280 includes P-Pinterfaces 1286 and 1288. Processors 1270, 1280 may exchange informationvia a point-to-point (P-P) interface 1250 using P-P interface circuits1278, 1288. As shown in FIG. 12, IMCs 1272 and 1282 couple theprocessors to respective memories, namely a memory 1232 and a memory1234, which may be portions of main memory locally attached to therespective processors.

Processors 1270, 1280 may each exchange information with a chipset 1290via individual P-P interfaces 1252, 1254 using point to point interfacecircuits 1276, 1294, 1286, 1298. Chipset 1290 may optionally exchangeinformation with the coprocessor 1238 via a high-performance interface1239. In one embodiment, the coprocessor 1238 is a special-purposeprocessor, such as, for example, a high-throughput MIC processor, anetwork or communication processor, compression engine, graphicsprocessor, GPGPU, embedded processor, or the like.

A shared cache (not shown) may be included in either processor oroutside of both processors, yet connected with the processors via P-Pinterconnect, such that either or both processors' local cacheinformation may be stored in the shared cache if a processor is placedinto a low power mode.

Chipset 1290 may be coupled to a first bus 1216 via an interface 1296.In one embodiment, first bus 1216 may be a Peripheral ComponentInterconnect (PCI) bus, or a bus such as a PCI Express bus or anotherthird generation I/O interconnect bus, although the scope of the presentinvention is not so limited.

As shown in FIG. 12, various I/O devices 1214 may be coupled to firstbus 1216, along with a bus bridge 1218 which couples first bus 1216 to asecond bus 1220. In one embodiment, one or more additional processor(s)1215, such as coprocessors, high-throughput MIC processors, GPGPU's,accelerators (such as, e.g., graphics accelerators or digital signalprocessing (DSP) units), field programmable gate arrays, or any otherprocessor, are coupled to first bus 1216. In one embodiment, second bus1220 may be a low pin count (LPC) bus. Various devices may be coupled toa second bus 1220 including, for example, a keyboard and/or mouse 1222,communication devices 1227 and a storage unit 1228 such as a disk driveor other mass storage device which may include instructions/code anddata 1230, in one embodiment. Further, an audio I/O 1224 may be coupledto the second bus 1220. Note that other architectures are possible. Forexample, instead of the point-to-point architecture of FIG. 12, a systemmay implement a multi-drop bus or other such architecture.

Referring now to FIG. 13, shown is a block diagram of a second morespecific exemplary system 1300 in accordance with an embodiment of thepresent invention. Like elements in FIGS. 12 and 13 bear like referencenumerals, and certain aspects of FIG. 12 have been omitted from FIG. 13in order to avoid obscuring other aspects of FIG. 13.

FIG. 13 illustrates that the processors 1270, 1280 may includeintegrated memory and I/O control logic (“CL”) 1272 and 1282,respectively. Thus, the CL 1272, 1282 include integrated memorycontroller units and include I/O control logic. FIG. 13 illustrates thatnot only are the memories 1232, 1234 coupled to the CL 1272, 1282, butalso that I/O devices 1314 are also coupled to the control logic 1272,1282. Legacy I/O devices 1315 are coupled to the chipset 1290.

Referring now to FIG. 14, shown is a block diagram of a SoC 1400 inaccordance with an embodiment of the present invention. Similar elementsin FIG. 10 bear like reference numerals. Also, dashed lined boxes areoptional features on more advanced SoCs. In FIG. 14, an interconnectunit(s) 1402 is coupled to: an application processor 1410 which includesa set of one or more cores 202A-N and shared cache unit(s) 1006; asystem agent unit 1010; a bus controller unit(s) 1016; an integratedmemory controller unit(s) 1014; a set or one or more coprocessors 1420which may include integrated graphics logic, an image processor, anaudio processor, and a video processor; an static random access memory(SRAM) unit 1430; a direct memory access (DMA) unit 1432; and a displayunit 1440 for coupling to one or more external displays. In oneembodiment, the coprocessor(s) 1420 include a special-purpose processor,such as, for example, a network or communication processor, compressionengine, GPGPU, a high-throughput MIC processor, embedded processor, orthe like.

Embodiments of the mechanisms disclosed herein may be implemented inhardware, software, firmware, or a combination of such implementationapproaches. Embodiments of the invention may be implemented as computerprograms or program code executing on programmable systems comprising atleast one processor, a storage system (including volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device.

Program code, such as code 1230 illustrated in FIG. 12, may be appliedto input instructions to perform the functions described herein andgenerate output information. The output information may be applied toone or more output devices, in known fashion. For purposes of thisapplication, a processing system includes any system that has aprocessor, such as, for example; a digital signal processor (DSP), amicrocontroller, an application specific integrated circuit (ASIC), or amicroprocessor.

The program code may be implemented in a high level procedural or objectoriented programming language to communicate with a processing system.The program code may also be implemented in assembly or machinelanguage, if desired. In fact, the mechanisms described herein are notlimited in scope to any particular programming language. In any case,the language may be a compiled or interpreted language.

One or more aspects of at least one embodiment may be implemented byrepresentative instructions stored on a machine-readable medium whichrepresents various logic within the processor, which when read by amachine causes the machine to fabricate logic to perform the techniquesdescribed herein. Such representations, known as “IP cores” may bestored on a tangible, machine readable medium and supplied to variouscustomers or manufacturing facilities to load into the fabricationmachines that actually make the logic or processor.

Such machine-readable storage media may include, without limitation,non-transitory, tangible arrangements of articles manufactured or formedby a machine or device, including storage media such as hard disks, anyother type of disk including floppy disks, optical disks, compact diskread-only memories (CD-ROMs), compact disk rewritable' s (CD-RWs), andmagneto-optical disks, semiconductor devices such as read-only memories(ROMs), random access memories (RAMs) such as dynamic random accessmemories (DRAMs), static random access memories (SRAMs), erasableprogrammable read-only memories (EPROMs), flash memories, electricallyerasable programmable read-only memories (EEPROMs), phase change memory(PCM), magnetic or optical cards, or any other type of media suitablefor storing electronic instructions.

Accordingly, embodiments of the invention also include non-transitory,tangible machine-readable media containing instructions or containingdesign data, such as Hardware Description Language (HDL), which definesstructures, circuits, apparatuses, processors and/or system featuresdescribed herein. Such embodiments may also be referred to as programproducts.

Emulation (Including Binary Translation, Code Morphing, etc.)

In some cases, an instruction converter may be used to convert aninstruction from a source instruction set to a target instruction set.For example, the instruction converter may translate (e.g., using staticbinary translation, dynamic binary translation including dynamiccompilation), morph, emulate, or otherwise convert an instruction to oneor more other instructions to be processed by the core. The instructionconverter may be implemented in software, hardware, firmware, or acombination thereof. The instruction converter may be on processor, offprocessor, or part on and part off processor.

FIG. 15 is a block diagram contrasting the use of a software instructionconverter to convert binary instructions in a source instruction set tobinary instructions in a target instruction set according to embodimentsof the invention. In the illustrated embodiment, the instructionconverter is a software instruction converter, although alternativelythe instruction converter may be implemented in software, firmware,hardware, or various combinations thereof. FIG. 15 shows a program in ahigh level language 1502 may be compiled using an x86 compiler 1504 togenerate x86 binary code 1506 that may be natively executed by aprocessor with at least one x86 instruction set core 1516. The processorwith at least one x86 instruction set core 1516 represents any processorthat can perform substantially the same functions as an Intel processorwith at least one x86 instruction set core by compatibly executing orotherwise processing (1) a substantial portion of the instruction set ofthe Intel x86 instruction set core or (2) object code versions ofapplications or other software targeted to run on an Intel processorwith at least one x86 instruction set core, in order to achievesubstantially the same result as an Intel processor with at least onex86 instruction set core. The x86 compiler 1504 represents a compilerthat is operable to generate x86 binary code 1506 (e.g., object code)that can, with or without additional linkage processing, be executed onthe processor with at least one x86 instruction set core 1516.Similarly, FIG. 15 shows the program in the high level language 1502 maybe compiled using an alternative instruction set compiler 1508 togenerate alternative instruction set binary code 1510 that may benatively executed by a processor without at least one x86 instructionset core 1514 (e.g., a processor with cores that execute the MIPSinstruction set of MIPS Technologies of Sunnyvale, Calif. and/or thatexecute the ARM instruction set of ARM Holdings of Sunnyvale, Calif.).The instruction converter 1512 is used to convert the x86 binary code1506 into code that may be natively executed by the processor without anx86 instruction set core 1514. This converted code is not likely to bethe same as the alternative instruction set binary code 1510 because aninstruction converter capable of this is difficult to make; however, theconverted code will accomplish the general operation and be made up ofinstructions from the alternative instruction set. Thus, the instructionconverter 1512 represents software, firmware, hardware, or a combinationthereof that, through emulation, simulation or any other process, allowsa processor or other electronic device that does not have an x86instruction set processor or core to execute the x86 binary code 1506.

In the description and claims, the term “logic” may have been used. Asused herein, the term logic may include but is not limited to hardware,firmware, software, or a combination thereof. Examples of logic includeintegrated circuitry, application specific integrated circuits, analogcircuits, digital circuits, programmed logic devices, memory devicesincluding instructions, etc. In some embodiments, the logic may includetransistors and/or gates potentially along with other circuitrycomponents.

In the description and claims, the terms “coupled” and “connected,”along with their derivatives, may have been used. It should beunderstood that these terms are not intended as synonyms for each other.Rather, in particular embodiments, “connected” may be used to indicatethat two or more elements are in direct physical or electrical contactwith each other. “Coupled” may mean that two or more elements are indirect physical or electrical contact. However, “coupled” may also meanthat two or more elements are not in direct contact with each other, butyet still co-operate or interact with each other.

The term “and/or” may have been used. As used herein, the term “and/or”means one or the other or both (e.g., A and/or B means A or B or both Aand B).

In the description above, for the purposes of explanation, numerousspecific details have been set forth in order to provide a thoroughunderstanding of the embodiments of the invention. It will be apparenthowever, to one skilled in the art, that one or more other embodimentsmay be practiced without some of these specific details. The particularembodiments described are not provided to limit the invention but toillustrate it. The scope of the invention is not to be determined by thespecific examples provided above but only by the claims below. Allequivalent relationships to those illustrated in the drawings anddescribed in the specification are encompassed within embodiments of theinvention. In other instances, well-known circuits, structures, devices,and operations have been shown in block diagram form or without detailin order to avoid obscuring the understanding of the description.

Where considered appropriate, reference numerals have been repeatedamong the figures to indicate components that may optionally besubstantially the same and have similar characteristics. In otherinstances, terminal portions of reference numerals have been repeatedamong the figures to indicate corresponding or analogous elements, whichmay optionally have similar or the same characteristics unless specifiedor clearly apparent otherwise. In some cases, where multiple componentshave been shown and described, they may be incorporated into a singlecomponent. In other cases, where a single component has been shown anddescribed, it may be separated into two or more components. In thedrawings, arrows represent couplings and bidirectional arrows representbidirectional couplings.

Various operations and methods have been described. Some of the methodshave been described in a relatively basic form in the flow diagrams, butoperations may optionally be added to and/or removed from the methods.In addition, while the flow diagrams show a particular order of theoperations according to example embodiments, it is to be understood thatthat particular order is exemplary. Alternate embodiments may optionallyperform the operations in different order, combine certain operations,overlap certain operations, etc. The components, features, and specificoptional details described herein for an apparatus also optionally applyto the methods described herein which may in embodiments be performed byand/or with such an apparatus.

Some embodiments include an article of manufacture (e.g., a computerprogram product) that includes a machine-readable medium. The medium mayinclude a mechanism that provides, for example stores, information in aform that is readable by the machine. The machine-readable medium mayprovide, or have stored thereon, one or more, or a sequence ofinstructions, that if executed by a machine causes the machine toperform and/or results in the machine performing one or operations,methods, or techniques disclosed herein. In some embodiments, themachine-readable medium may include a tangible non-transitorymachine-readable storage media. For example, the tangible non-transitorymachine-readable storage media may include a floppy diskette, an opticalstorage medium, an optical disk, a CD-ROM, a magnetic disk, amagneto-optical disk, a read only memory (ROM), a programmable ROM(PROM), an erasable-and-programmable ROM (EPROM), anelectrically-erasable-and-programmable ROM (EEPROM), a random accessmemory (RAM), a static-RAM (SRAM), a dynamic-RAM (DRAM), a Flash memory,a phase-change memory, or the like. The tangible medium may include oneor more solid or tangible physical materials, such as, for example, asemiconductor material, a phase change material, a magnetic material,etc. In another embodiment, the machine-readable media may include anon-tangible transitory machine-readable communication medium, forexample, the electrical, optical, acoustical or other forms ofpropagated signals, such as carrier waves, infrared signals, digitalsignals, etc.

Examples of suitable machines include, but are not limited to, computersystems, desktops, laptops, notebooks, netbooks, nettops, MobileInternet devices (MIDs), servers, network elements (e.g., routers,switches, etc.) cellular phones, media players, nettops, set-top boxes,video game controllers, and other electronic devices having one or moreprocessors or other instruction execution apparatus. Such electronicdevices typically include one or more processors coupled with one ormore other components, such as one or more storage devices(non-transitory machine-readable storage media), user input/outputdevices (e.g., a keyboard, a touchscreen, and/or a display), and/ornetwork connections. The coupling of the processors and other componentsis typically through one or more busses and bridges (also termed buscontrollers). Thus, the storage device of a given electronic device maystore code and/or data for execution on the one or more processors ofthat electronic device. Alternatively, one or more parts of anembodiment of the invention may be implemented using differentcombinations of software, firmware, and/or hardware.

It should also be appreciated that reference throughout thisspecification to “one embodiment”, “an embodiment”, or “one or moreembodiments”, for example, means that a particular feature may beincluded in the practice of the invention. Similarly, it should beappreciated that in the description various features are sometimesgrouped together in a single embodiment, Figure, or description thereoffor the purpose of streamlining the disclosure and aiding in theunderstanding of various inventive aspects. This method of disclosure,however, is not to be interpreted as reflecting an intention that theinvention requires more features than are expressly recited in eachclaim. Rather, as the following claims reflect, inventive aspects maylie in less than all features of a single disclosed embodiment. Thus,the claims following the Detailed Description are hereby expresslyincorporated into this Detailed Description, with each claim standing onits own as a separate embodiment of the invention.

What is claimed is:
 1. A processor comprising: a plurality of cores toexecute instructions and process data, the plurality of cores comprisingsymmetric multi-threaded cores; a cache subsystem to store theinstructions and data, the cache subsystem comprising a plurality offirst-level caches and at least one higher-level distributed cachecomprising a plurality of cache portions shared by the plurality ofcores, each first-level cache integral to one of the plurality of coresand each cache portion accessible to each of the plurality of cores;cache management circuitry to provide coherent, non-uniform access tothe plurality of cache portions by the plurality of cores; and powermanagement circuitry to enable a first frequency to be set for a firstsubset of the plurality of cores and a second frequency to be set for asecond subset of the plurality of cores, the power management circuitryto selectively gate power to the first subset of the plurality of cores,the second subset of the plurality of cores, and/or one or more of thecache portions.
 2. The processor of claim 1 wherein each cache portionis physically closer to at least one subset of the plurality of coresthan it is to other of the plurality of cores.
 3. The processor of claim1 wherein the power management circuitry is to selectively gate power inaccordance with a detected workload on the first and/or second subsetsof the plurality of cores.
 4. The processor of claim 1 wherein a firstcore of a first predetermined subset is physically proximate to a firstcache portion.
 5. The processor of claim 4 further comprising: a localcore-cache interconnect to couple the first core to the first cacheportion; and an on-die interconnect to couple one or more other cores tothe first cache portion.
 6. The processor of claim 5 wherein the localcore-cache interconnect and on-die interconnect provide different accesslatency and/or data throughput to a given cache portion.
 7. Theprocessor of claim 5 wherein the local core-cache interconnect is toprovide relatively lower latency and/or higher data throughput to thefirst core when the first core accesses the first cache portion than theon-die interconnect is to provide to the one or more other cores whenthe one or more other cores access the first cache portion.
 8. Theprocessor of claim 1 wherein, based on a workload on each of the firstand second subsets of the plurality of cores, a thread is to beselectively migrated from the first subset to the second subset.
 9. Theprocessor of claim 1 wherein at least one of the plurality of corescomprises: an instruction fetch circuit to fetch instructions from acorresponding first level cache or a system memory; and a decode circuitto decode the instructions; and an execution circuit comprising aplurality of functional units to execute the instructions.
 10. Theprocessor of claim 1 wherein the first subset of the plurality of coresform a first virtual processor and the second subset of the plurality ofcores form a second virtual processor.
 11. The processor of claim 10wherein the execution circuit is to execute instructions to implement avirtual machine monitor (VMM), the VMM to form the first virtualprocessor by combining the first subset of the plurality of cores and toform the second virtual processor by combining the second subset of theplurality of cores.
 12. The processor of claim 1 wherein the non-uniformaccess to the plurality of cache portions by the plurality of cores isbased, at least in part, on each cache portion being coupled to aproximate subset of the plurality of cores over a first interconnecthaving first characteristics and being coupled to a non-proximate subsetof the plurality of cores over a second interconnect having secondcharacteristics.
 13. A method comprising: executing instructions andprocessing data with a plurality of cores, the plurality of corescomprising symmetric multi-threaded cores; storing the instructions andthe data in a cache subsystem, the cache subsystem comprising aplurality of first-level caches and at least one higher-leveldistributed cache comprising a plurality of cache portions, eachfirst-level cache integral to one of the plurality of cores and eachcache portion accessible to each of the plurality of cores; sharing theplurality of cache portions among the plurality of cores; providingcoherent, non-uniform access to the plurality of cache portions by theplurality of cores; enabling a first frequency to be set for a firstsubset of the plurality of cores and a second frequency to be set for asecond subset of the plurality of cores; and selectively gating power tothe first subset of the plurality of cores, the second subset of theplurality of cores, and/or one or more of the cache portions.
 14. Themethod of claim 13 wherein said selectively gating the power comprisesselectively gating the power in accordance with a detected workload onthe first and/or second subsets of the plurality of cores.
 15. Themethod of claim 13 further comprising selectively migrated a thread fromthe first subset to the second subset based on a workload on each of thefirst and second subsets of the plurality of cores.
 16. The method ofclaim 13 further comprising forming the first subset of the plurality ofcores as a first virtual processor and forming the second subset of theplurality of cores as a second virtual processor.
 17. The method ofclaim 16 further comprising executing instructions to implement avirtual machine monitor (VMM), the VMM to form the first virtualprocessor by combining the first subset of the plurality of cores and toform the second virtual processor by combining the second subset of theplurality of cores.
 18. The method of claim 13 wherein said providingthe non-uniform access to the plurality of cache portions by theplurality of cores is based, at least in part, on each cache portionaccessing a proximate subset of the plurality of cores over a firstinterconnect having first characteristics and accessing a non-proximatesubset of the plurality of cores over a second interconnect havingsecond characteristics.
 19. A machine comprising: a processorcomprising: a plurality of cores to execute instructions and processdata, the plurality of cores comprising symmetric multi-threaded cores;a cache subsystem to store the instructions and data, the cachesubsystem comprising a plurality of first-level caches and at least onehigher-level distributed cache comprising a plurality of cache portionsshared by the plurality of cores, each first-level cache integral to oneof the plurality of cores and each cache portion accessible to each ofthe plurality of cores; and cache management circuitry to providecoherent, non-uniform access to the plurality of cache portions by theplurality of cores; and an article of manufacture comprising amachine-readable storage medium storing a plurality of instructions, theplurality of instructions, if performed by a machine, to cause themachine to perform operations comprising to: enable a first frequency tobe set for a first subset of the plurality of cores and a secondfrequency to be set for a second subset of the plurality of cores; andselectively gate power to the first subset of the plurality of cores,the second subset of the plurality of cores, and/or one or more of thecache portions.
 20. The machine of claim 19 wherein the instructionsfurther comprise instructions that if performed by the machine are tocause the machine to form the first subset of the plurality of cores asa first virtual processor and form the second subset of the plurality ofcores as a second virtual processor.